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REBUILDING RELATIONSHIPS 
BETWEEN DATA, METHOD, 
AND THEORIES 


How the Scientific Method Can Help 


Jeffrey M. Cucina and Mary Anne Nester 


When authors write manuscripts for industrial-organizational (I-O) psychol- 
ogy and management literatures, they are strongly encouraged to make a new 
theoretical contribution if they want their work to be published in the pre- 
miere outlets. This is a relatively recent trend in I-O psychology, as docu- 
mented by Cucina and Moriarty (2015) who showed a dramatic increase in the 
use of words beginning with “theor” over the history of the Journal of Applied 
Psychology (JAP) and Personnel Psychology (PPsych). Editorial statements and 
manuscript criteria also changed over time and authors have responded by pro- 
posing lengthy theoretical models and discourse in their manuscripts. Many 
I-O psychologists have noted and debated the rise of “theory” in the field 
(Aguinis, Bradley & Broderson, 2014; Campbell & Wilmot, 2018; Cortina, 
2014; Cucina et al., 2015; Kepes & McDaniel, 2013; Kohler, DeSimone & 
Schoen, 2020; Ryan & Ployhart, 2014). In the past, theoretical work in I-O 
psychology was considered the exception rather than the rule, with a greater 
focus on empirical data. 

The emphasis on theory has a longer history within the management litera- 
ture. Indeed, there is speculation that the increased role of theory in I-O psy- 
chology stems from the management literature (Aguinis et al., 2014). Journals 
in management, such as the Academy of Management Journal (AMJ), often require 
manuscripts to make a theoretical contribution and will even desk reject manu- 
scripts that do not emphasize theory. Note that Sutton and Staw (1995) specifi- 
cally singled out I-O psychology's JAP and PPsych as being at the “most empirical 
end of the spectrum” of management-related journals (p. 379). These journals 
have changed and now are more in line with management journals, such as AMJ, 
which require strong theoretical contributions to accompany empirical research. 


DOI: 10.4324/9781003015000-17 


318 J. M. Cucina and M. A. Nester 


At the same time that journals have increased emphasis on theory, data scientists 
and some I-O psychologists have begun using machine learning in applied set- 
tings, which is a methodology that largely relies on what was once known as 
“dustbowl empiricism.” This has a resulted in a disjointed relationship between 
data, methods, and theory, with some I-O psychologists overemphasizing the- 
ory, others focusing on data at the expense of building scientific knowledge, and 
a gap between statistical advances and research practice. 

There is a need in I-O psychology to rebuild the relationships between data, 
methods, and theory, and the aim of this chapter is to explain how these con- 
cepts should be related. We use the scientific method as a framework for this 
discussion. In the remainder of this chapter, we review the scientific method, 
describe how data, methods, and theory are related within the framework 
of the scientific method, and discuss two other configurations and research 
approaches. 


Overview of the Scientific Method 


Organizational researchers are no doubt familiar with the spirit and conduct 
of science. However, we view it as instructive to review the scientific method 
as it is defined and implemented in other fields of science because the recent 
emphasis on theory building and theoretical contributions in I-O psychology 
represents a significant philosophical departure from the scientific method. For 
instance, authors of scientific papers in I-O psychology are often expected to 
start a paper or research project by developing theory, often devoting significant 
journal space to theory exposition. There is also an expectation that the pro- 
posed theory will be supported, which results in theory-reafirming data and 
results. As a result, there is less emphasis on important empirical work, studies 
that report interesting and practically useful results (but that lack theory devel- 
opment), and research in areas that do not have extensive and complex theories. 
In contrast, the scientific method and research practices in other disciplines often 
do not begin with theory but instead with an observation. The scientific method 
also allows for research designs that can disconfirm (as well as confirm) proposed 
hypotheses and theories; in fact, there is an emphasis on designing studies that 
would disconfirm a theory. 

Many other areas of science have a formally described approach to conduct- 
ing scientific research. This approach consists of an iterative seven-step pro- 
cess, which Cucina et al. (2014) observed to recur in their review of textbooks 
and coursework from various scientific fields (e.g., astronomy, physics, biology, 
chemistry). A depiction of the scientific method is provided in Figure 13.1. In 
the following sections, we describe each step in detail and explain how data, methods, 
and theory relate to the activities within each step. 
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Step 1: Make an observation. 


Step 2: Form a research question. 


Step 3: Write a hypothesis 


Step 4: Make a prediction 


Step 5: Test the hypothesis using experiment 
or observation 


Step 6a: If the hypothesis is 
supported, then make new 
tests for the hypothesis 


Step 6b: If the hypothesis is 
not supported, then revise 
or create new a hypothesis 


Step 8: After many successful replications, 


Step 7: Repeat steps 4 through 6 many times 


establish a theory 


FIGURE 13.1 The scientific method as summarized by Cucina et al. (2014). 


Step 1: Make an Observation 


The scientific method begins with an observation about the natural world 
(including organizations and employees). An observation could be an anec- 
dote, a casual observance of the everyday world, the results of previous scientific 
research, a case study, or information gleaned when providing consulting services 
to organizations. For example, a researcher could simply observe that individuals 
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in a certain village have a lower incidence of a certain disease. Alternately, a 
researcher with access to a large dataset might observe a correlation between two 
variables. An organizational researcher might make an observation by watching 
the interaction between a leader and a subordinate. An observation could even 
be the results of a previously published study. 

Although observations are most often based on data, observations based on 
methods and theory could serve as the basis for step 1 of the scientific method. 
The observations mentioned in the previous paragraph are essentially data-based 
observations. In some cases, the quality of the data is not robust, with small 
samples, uncontrolled environments, and measures with questionable construct 
validity limiting the conclusions that can be drawn from the study. However, 
this is not a problem as more robust testing of the hypothesis will occur in 
the subsequent steps. The observation in step 1 could also be based on more 
robust data (even published data and meta-analyses). For instance, a researcher 
might notice the correlation between two variables in a meta-analysis and build 
follow-up hypotheses. Observations can also be made about methods used in 
previous studies and a researcher could create hypotheses about using other 
methodologies to study an issue. Observations can also be based on existing 
theories, including conflicts between competing theories, or the applicability 
of existing theories to new areas. Observations about a stream of research can 
also be made. For instance, problematization (e.g., identifying incompleteness, 
inadequacy, and incommensurability in existing research as described by Locke 
& Golden-Biddle, 1997) can also serve as the basis for observations. An observa- 
tion could also be based on issues facing organizations. 

There are also circumstances in which a combination of data, methods, and 
theory can serve as the basis for an observation. For instance, Schmidt and 
Hunter’s (2003) development of validity generalization and meta-analysis was 
initially built on data-based observations that showed differing validity coef- 
ficients for different industry jobs versus military data that showed consistent 
validity coefficients, a method-based observation that industrial studies often had 
much smaller sample sizes and much larger sampling error versus military stud- 
ies, and a theoretical observation concerning the theory of situational specificity. 
This led to a hypothesis that the industrial data was inconsistent across studies 
due to sampling error. This was followed by the subsequent development of 
meta-analytic techniques, which in turn led to the theory of validity generaliza- 
tion and the disconfirmation of situational specificity theory. 


Step 2: Form a Research Question 


Most research questions in the organizational sciences focus on the relationship 
(e.g., correlation, regression, mean differences) between two (or more) natu- 
rally occurring variables. The research question should be concise and easily 
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understood. Some examples include “does spatial reasoning correlate with job 
performance for pilots?” “are job satisfaction and turnover correlated?” and 
“which tasks and competencies are important for a particular job?” At this point 
in the process, it can be helpful to consider how meaningful and important 
the research question is for the field of study (e.g., the potential relevance to 
organizations). 


Step 3: Write a Hypothesis 


A hypothesis is a broad statement that aims to answer the research question. It 
need not be elaborately thought out; an “educated guess” (Cucina et al., 2014, 
pp. 358-359), conjecture, or even a hunch can serve as a hypothesis. As we will 
discuss later, the hypothesis could very well be revised or discarded in later steps. 
The hypothesis should be written so that it can apply to multiple studies, not just 
the study at hand. In some ways, the new theory that many I/O psychology and 
management journals ask authors to create in an introduction section is similar 
to a hypothesis. However, hypotheses are different in several regards. A hypoth- 
esis should be clear, concise, and elegant. It should not be lengthy with a large 
number of variables and many proposed relationships. In principle, it should be 
fully testable in one paper. A hypothesis could be based on existing theory, but 
it could also be based on an entirely new development. 

At this point in the scientific method, it is beneficial to evaluate the hypoth- 
esis and potentially revise it before proceeding further. A hypothesis should 
make a testable proposition and it should be falsifiable (Popper, 1934, 1959). 
The prediction can be predicated on the logic that if the hypothesis is true, then 
the prediction must be true. However, Platt (1964) also makes a case for “strong 
inference” whereby a prediction is made such that if the prediction is true, then 
the hypothesis must be false. Ideally, effort should be given to the development 
of competing hypotheses for the research question, which would allow use of 
Platt’s (1964) strong inference, which will be mentioned in the subsequent steps. 
Some elements of problematization (Locke & Golden-Biddle, 1997), especially 
incommensurability problematization (in which alternate theses are explored 
and assumptions underlying a particular area of research are examined and chal- 
lenged) are also relevant here. 


Step 4: Make a Prediction 


A prediction is more specific in scope than a hypothesis because it is written to apply 
to a single study, not multiple studies. It focuses more on what is expected in the 
specific context of a study in terms of the methodology (e.g., participant popula- 
tion, organizational context, measures) rather being a general statement, such as the 
hypothesis. It also specifies the expected results of the data analysis. A prediction 
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can often be portrayed formulaically (e.g., Mea roup a > Mean cup p° with a 
Cohen’s, 1992, d near 1.0). The prediction is written to apply to the methodol- 
ogy of the study, including the measures used, the experimental manipulations, 


and the operational definitions of the variables. 


Step 5: Test the Hypothesis Using Experiment or Observation 


In this step, an empirical study is conducted to test the hypothesis. Although 
there are aspects of theory that relate to this step (e.g., psychometric and statisti- 
cal theory are employed), data and methods play the key roles. The researcher 
must first design an empirical study and some thought needs to be given to 
the type of data that will be collected, the suitability of the data for testing the 
hypothesis, and the data analysis strategy. The methods used in the study are 
crucial in yielding suitable data for hypothesis testing. If competing hypotheses 
were generated in step 3, the methodology of the empirical study should yield 
data that could rule out one or more of the hypotheses, allowing for a “crucial 
experiment” (Platt, 1964, p. 347). 

It can be helpful to review data from previous studies concerning the meth- 
ods and data quality. If there are flaws in the research methods or the data is of 
poor quality, then the experiment or observational study may not adequately 
test the hypothesis. For psychologists, issues such as construct validity and scale 
reliability are critical. Many psychological measures, especially behavioral ones, 
are inherently unreliable. For instance, supervisory ratings of job performance 
have interrater reliabilities of about .52 (Shen et al., 2014; Viswesvaran, Ones & 
Schmidt, 1996). Many of the behavioral outcome variables used to study the 
relationship between personality and behavior were based on unreliable single 
instances of behavior. This led Mischel (1968) to question the importance of 
personality (vs. situational variables) in determining behavior, which had theo- 
retical implications for personality and social psychology. However, a rebuttal 
by Epstein (1979) demonstrated that personality predicts behavior when more 
reliable behavioral outcome variables based on multiple instances of behavior 
are created. Thus, when designing a study, the potential validity and reliability 
of the measures used should be considered, lest a researcher could erroneously 
conclude that they have discovered a new construct or that their construct does 
not predict behavior. 

In addition, there are other methodological considerations. For example, a 
power analysis should determine the sample size; however, consideration of the 
accuracy of effect size point estimates should also be considered. For instance, 
suppose a researcher anticipates a correlation of .30 between two variables. A 
total of 115 cases will yield 95% power for detecting the correlation; however, 
the 95% confidence interval about a correlation of .3 with 115 cases ranges from 
.12 to .46 (which is a large interval). The situation becomes worse when artifacts 
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such as measurement error and range restriction are taken into account (Schmidt 
etal., 1976). Unfortunately, in our experience as reviewers of many I-O psychol- 
ogy manuscripts and presentations, empirical studies are too often conducted 
with measures having questionable construct validity, poor reliability, insufficient 
power. and insufficient precision in effect size estimates. This is unfortunate 
given the researcher and participant time and effort involved in conducting many 
empirical studies. 

Once the empirical study is completed, the data are analyzed to determine if 
the hypothesis was supported. Obviously, quantitative and qualitative methods 
play an important role in this phase. Sometimes, data analysis can shed new insight 
on the research question and hypothesis, leading to further refinements of the 
hypothesis or development of post hoc hypotheses, as will be discussed in step 6b. 


Step 6a: If the Hypothesis Is Supported, Then Make 
New Tests for the Hypothesis 


This step is quite similar to step 5; however, researchers should attempt to test 
the hypothesis using additional empirical studies. Step 6a, can be implemented 
with multi-study articles or with separate follow-up studies (although we would 
recommend avoiding piecemeal publication). The new studies could be con- 
ducted by different research teams (e.g., a second research team might design a 
new test of a hypothesis that another team studied previously). The use of addi- 
tional studies is important because Type I and II errors, unknown confounds, 
and experimenter error can lead to incorrect conclusions about hypotheses. 

Direct replication is one approach to additional testing (Open Science 
Collaboration, 2015); however, modified conceptual replication often provides 
a better test of the hypothesis (Cucina & Hayes, 2015). In fact, modified con- 
ceptual replications are more consistent with the spirit of the scientific method 
since these allow for more robust testing of the hypothesis using different meth- 
odologies, types of data, measures, manipulations, populations, and so forth. 
For instance, directly replicating a criterion-related validation study of a scale 
measuring a new construct which predicted cashier job performance in a second 
sample of cashiers does not yield as much scientific information as a conceptual 
replication would. A conceptual replication might compare other approaches for 
measuring the construct, how well the construct adds incremental validity over 
other tests, whether the construct predicts performance for other jobs and for 
other domains of performance. Thus, in our view, parts of the recent “replica- 
tion crisis” in psychology have been misguided given that an important goal for 
science is to develop a database of existing studies containing many different 
types of tests for a hypothesis. This allows for an evaluation of the external and 
ecological validity of the initial study’s conclusion and a better determination of 
whether moderators exist. 
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Step 6b: If the Hypothesis Is Not Supported, 
Then Revise or Create a New Hypothesis 


When a hypothesis is not supported, the best approach is to change the hypoth- 
esis, provided that the methodology and data used to test it were sufficient. This 
can be accomplished by revising the hypothesis or discarding it in favor of a new 
hypothesis. This is an instance in which data, and to some extent methods (e.g., 
if use of a particular methodology suggests that the hypothesis is not supported) 
inform a hypothesis. 

In many fields, including I-O psychology, researchers are expected to predict 
the outcome of their studies correctly before collecting and analyzing their data. 
Publishing a study that does not support an initial hypothesis is discouraged and, 
in some cases, forbidden by the policies of academic journals (Cortina, 2016). 
Additionally researchers cannot revise their hypotheses (step 6b), unless they 
resort to hypothesizing after the results are known (HARKing, Kerr, 1998) or 
collecting additional data and running statistical tests multiple times to achieve 
significance (Simmons, Nelson & Simonsohn, 2011). There is evidence that 
organizational researchers sometimes alter their hypotheses after the data is col- 
lected (O’Boyle, Banks & Gonzalez-Mulé, 2017). When unreported in manu- 
scripts, both practices are disingenuous and create statistical issues. 

A much better approach is to simply allow researchers to admit their hypoth- 
eses are wrong in their papers’ discussion sections and to propose (and possibly 
test in follow-up studies) new post hoc hypotheses in an open and transparent 
manner. Results-blind manuscript reviews and registered reports (which some 
I-O psychology outlets such as the Journal of Business and Psychology are now sup- 
porting) can serve as a foundation for this practice. Giving authors allowances 
(perhaps in supplemental materials) to describe additional exploratory analyses 
they conducted on the data can also assist in increasing transparency and pro- 
viding better documentation of the research study. Additionally, the practice of 
maintaining a laboratory notebook? (Pain, 2019) often seen in the natural sci- 
ences, and open notebook science (Bradley, 2006, 2007; Schapira et al., 2019) 
could encourage researchers to be more transparent and provide more valuable 
information about their data analyses. 

Many of science’s greatest discoveries began with a researcher who first made 
a hypothesis that turned out to be flatly incorrect but later made and tested a 
new hypothesis that turned out to be correct. As described in Watson’s (1997) 
firsthand account, Watson and Crick (1953) went through multiple iterations of 
hypotheses for the structure of DNA. They hypothesized that DNA consisted of 
1, 2, 3, or 4 chains, that the chains were held together by Magnesium or Calcium 
ions (in fact they are held together by hydrogen bonds), that DNA’s helix had 
rotations of different lengths (e.g., 28 or 68 Angstroms), and so forth. Their work 
led to so many disproven hypotheses that their laboratory director, Sir Francis 
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Bragg (a Nobel Laureate), attempted to ban them from continuing work on the 
topic. Efforts by other researchers, including Linus Pauling (Pauling & Corey, 
1953), also consisted of multiple tests of disproven hypotheses. The scientific 
method allows for this research strategy and has led to Nobel Prize winning 
discoveries like Watson and Crick. 


Step 7: Repeat Steps 4-6 Many Times 


The seventh step in the scientific method involves repeated testing of the 
hypothesis in different settings, using different methodologies, and often involv- 
ing multiple teams of researchers. Independent verification of findings and con- 
ceptual variations of the methodology are important practices for hypothesis 
testing. This helps to avoid situations in which a researcher is motivated to mar- 
shal support for a hypothesis that is credited to him or her, even ifa specific study 
does not demonstrate support for the hypothesis. 


Step 8: After Many Successful Replications, Establish a Theory 


A hypothesis can only become a theory if it is supported after multiple replica- 
tions (or if it can be shown to be true using logical or mathematical proofs). 
Whether a hypothesis rises to the level of a theory depends on the amount and 
quality of data from the studies that are used to test it as well as an evaluation of 
the adequacy of the methodology used to test it. 

Establishing a theory is a group effort undertaken by an entire scientific com- 
munity. It requires multiple replications in different settings to yield enough 
studies for a thorough meta-analysis. Even researchers who established theories 
without collecting their own data (e.g., Einstein’s work on general relativity, 
meta-analysts showing the validity generalization of conscientiousness and gen- 
eral mental ability) make use of data from multiple studies conducted by other 
researchers. 

Note that a single researcher would likely not follow all of the steps of the 
scientific method in a single study. Instead, he or she could begin with conjec- 
tures, observations, general conclusions, and other ideas and conduct and pub- 
lish research on different steps of the process. For instance, a researcher could 
develop hypotheses or publish observations that could later be followed up and 
tested empirically by other research teams. Further note that abduction De, 
proposing and testing hypotheses that explain a phenomenon) can also be imple- 
mented using the scientific method. Essentially, a researcher could use abduction 
to generate explanatory hypotheses which are then tested using the steps in the 
scientific method. 

As we will discuss later in this chapter, we believe that the current state of 
theory development in I-O psychology and management has diverged from how 
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theories are developed in other fields of science. I-O psychology and manage- 
ment researchers often start directly with creating a theory, ignoring the ear- 
lier steps in the scientific method that should lead to theory development. In 
addition, I-O psychology and management researchers are often encouraged to 
develop new theories in each paper they write as top journals often will not pub- 
lish papers that do not develop new theory or that seek to test existing theories 
and hypotheses (Hambrick, 2007). 


Designing Studies to Produce Quality Evidence 
and Evaluating the Quality of Evidence 


When designing tests of a hypothesis, a researcher should consider the quality of 
evidence that will be generated by the study and how that relates to the hypoth- 
esis. Researchers should also consider the quality of evidence in support (or not 
in support) of a hypothesis when deciding whether to revise it or create a new 
hypothesis and whether the hypothesis can become a theory. As many research- 
ers are aware, there are inherent issues with research practices and the literature, 
including lack of replications, fallacies (e.g., the jingle-jangle fallacy, the fallacy 
that correlation implies causation), and insufficient testing of hypotheses. Thus, 
it is important to consider the quality of evidence for a hypothesis or theory. 
Medical researchers have devised a pyramid or hierarchy of evidence used to 
evaluate evidence for medical hypotheses and theories as part of evidence-based 
medicine. This framework, although not without its critics (e.g., Blunt, 2015), 
can be used by I-O psychologists when following the steps in the scientific 
method. Indeed, I-O psychologists have begun to incorporate aspects of evi- 
dence-based medicine into I-O psychology and management research (Pfeffer & 
Sutton, 2006; Reay, Berta & Kohn, 2009). 

The pyramid of evidence can be traced to a report by the Canadian Task 
Force on the Periodic Health Examination (1979). Since that time, different 
adaptations of the pyramid have been developed. Blunt (2020) has catalogued 
195 versions as of August 2020, almost all of which appear in medical out- 
lets. Reay et al. (2019) presented an adaptation of the pyramid for management 
research. Using many of the pyramids that Blunt (2020) catalogued, as well as 
the one by Reay et al. (2019), we compiled the pyramid of evidence shown in 
Figure 13.2. We added a few sources of evidence that are more germane to I-O 
psychology using asterisks. 

As one progresses from the bottom of the pyramid of evidence to the top, the 
quality of evidence increases. At the very bottom of the pyramid are untested 
hypotheses and at the top of the pyramid are theories and solid empirical find- 
ings, often demonstrating evidence of causality. The pyramid shown in Figure 
13.2 contains some methodologies that are not widely used in I-O psychol- 
ogy (e.g., case control studies) but that perhaps could be useful approaches to 
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Meta- 
analyses; 
systematic 
reviews; multiple 
randomized control 

trials with similar 

results; validity 

generalization” 


Randomized control trials 
(experiments with random 
assignment) 


Quasi-experiments 


Series of observational studies” 


Cohort studies (prospective observational studies) 


Case-control studies (retrospective observational 
studies with matching) 


Single observational study; time series study 


Literature review of multiple case studies; series of case studies 
Single case study/report 


Opinions/advice from individual experts, committees of experts, and 
authorities; clinical experience; narrative reviews; ideas; first principles; 
narrative literature reviews; “mechanism based reasoning” (Mupepele et al., 
2016, p. 1297), “evidence from stakeholders” (Pennington et al., 2017, p. 60), 
in vitro and animal studies, pseudotheory”, speculation”, practitioner 
experience and lore”, new untested hypotheses’. 


Note: Additions we made ourselves are denoted with asterisks. 


FIGURE 13.2 The pyramid of evidence as summarized from multiple sources. 


studying the effects of organizational interventions when experiments are not 
feasible. We also should note that in some contexts (e.g., personnel selection), the 
ultimate goal is to establish valid prediction rather than causality and thus some 
forms of evidence are not always applicable. 


The Role of Pseudotheorizing Within the Pyramid of Evidence 


It is worth noting that the pseudotheorizing (i.e., creating an elaborate untested 
hypothesis; Cucina et al., 2014) that has become popular in I-O psychology and 
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management journals is not explicitly included in the medical versions of the 
pyramid of evidence. It is most closely related to “mechanism-based reason- 
ing” and “mechanistic reasoning” which Mupepele et al. (2016) included in the 
bottom layer of the pyramid. They define this type of reasoning as a statement 
that is not based on empirical data but instead is based on an inferential chain 
of mechanisms. Howick, Glasziou & Aronson, (2010) provide more insight on 
mechanistic reasoning in the medical field, where it is defined as an inferen- 
tial link between mechanisms and an outcome for a patient. A mechanism is a 
hypothesis, theory (Howick et al., 2010), or “nomological machine” (Cartwright, 
1999, p. 50) involving features or systems that have regular inputs and outputs, 
such as the heart. Essentially, mechanistic reasoning is conjecture that uses estab- 
lished findings and concepts to predict the outcome associated with an inter- 
vention. Howick et al. (2010) pointed out several examples of cases in which 
mechanistic reasoning led to incorrect conclusions (e.g., the famous author of 
childrearing advice books, Dr. Spock, used mechanistic reasoning to recom- 
mend that parents place their babies on their stomachs when sleeping to reduce 
the risk of babies choking on vomit and dying of sudden infant death syndrome). 
In our field, pseudotheorizing is not always used to justify organizational inter- 
ventions; however, we do note several parallels between the quality of resulting 
evidence and thinking processes of mechanistic reasoning and pseudotheoriz- 
ing. Both involve using the literature to make inferential leaps and create new 
propositions. This is not to say that mechanistic reasoning and hypothesizing 
should be discounted, as Howick et al. (2010) point out. It can serve as the basis 
for hypotheses that are later tested empirically and that ascend the pyramid of 
evidence. Mechanistic reasoning can also be more valid when each link in the 
inferential chain is tied to robust empirical evidence. 

Howick et al. (2010) also noted an issue with lengthier inferential chains that 
has parallels with a fact that is based on path analysis. They give an example of 
an input having five intermediate steps, each with an effect of .90 probability 
leading to an outcome. On the surface, one might expect that the input will lead 
to the outcome with high certainty; however, the final effect of the intervention 
on the outcome is .90°, which equals only .59. A similar situation can occur with 
path analysis and organizational pseudotheorizing. Suppose that an organizational 
intervention has a standardized path analysis coefficient of .50 with construct A, 
which in turn has a .50 coefficient with construct B, which in turn has a .50 coef- 
ficient with job performance and that there are no significant unmodeled paths 
(e.g., a direct path from A to job performance) in the model (see Figure 13.3). 
We might conclude that the intervention has a sizable effect on job performance 
because the standardized coefficients in each path are all large effect sizes (per 
Cohen, 1992). However, the path analysis tracing rule (Kenny, 1979, 2004) tells 
us that the correlation between the intervention and job performance is only 
5 X .5 X .5, which equals .125, a small effect. Unfortunately, authors often 
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Job 


Intervention 
Performance 


5x5 x 5 = .125 


FIGURE 13.3 Path analysis model showing the diminishing effect from an interven- 
tion to job performance when it passes through two mediating constructs. 


create hypotheses and pseudotheories involving these types of paths between 
constructs without realizing that the underlying multiplicative effect leads to 
small outcomes. 


How the Pyramid of Evidence Informs the Scientific Method 


The pyramid of evidence can inform the scientific method and there is an inher- 
ent reciprocal relationship between the two. Much of the evidence at the base 
of the pyramid (e.g., opinions, advice, experience, a case study) can serve as 
the observations in step 1 of the scientific method and can inspire the research 
questions and hypotheses in steps 2 and 3. Steps 4 and 5 are represented in the 
middle of the pyramid of evidence where single observational or experimental 
studies appear. As additional studies testing the hypothesis are conducted (i.e., 
steps 6 and 7), the evidence for a hypothesis is closer to the top. Only when the 
hypothesis has been supported by repeated testing can it be considered a theory. 

The quality of evidence for testing a hypothesis and establishing a theory 
depends on the methodology used to collect data in empirical studies. Poorer 
quality methodologies (e.g., case studies) appear at the bottom of the pyramid 
of evidence. Methodologies that show a correlation between a potential cause 
and outcome appear in the middle, with statistical control methods (e.g., case- 
control studies, cohort studies) bolstering the quality of evidence. Experimental 
methods are required for demonstrating causality and these methods appear at 
the top of the pyramid. One of goals of evidence-based medicine is to identify 
which treatments have a causal effect on a patient's outcome, thus experiments 
De, randomized control trials) are often considered the best type of primary 
study. However, issues of Type I and II errors and replicability, and other meth- 
odological issues (Cook & Campbell, 1979) can impact the results of experiments 
and observational studies. This is why the pyramid of evidence places repeated 
studies with similar results (which can include meta-analyses of those studies) 
at its pinnacle and the scientific method defines theory (its pinnacle finding) 
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as a hypothesis that has received support from multiple studies using different 
methodologies. Thus, in most fields of science, a meta-analytic review would 
form the basis of theoretical establishment, whereas some I-O psychologists have 
written that “a review or meta-analysis does not constitute good theory” (Klein 
& Zedeck, 2004, p. 932). 

The pyramid of evidence also adds systematic reviews, which are common 
in the medical literature, to its pinnacle. Uman (2011) provides an overview 
of medical systematic reviews, which we summarize here. Unlike a traditional 
narrative literature review, when conducting a systematic review, the authors 
search and locate articles that meet certain criteria (e.g., a specific population, 
specific key terms). This is similar to the literature search strategy that is used in 
meta-analyses. In contrast, traditional narrative reviews often do not include a 
detailed list of search criteria, but instead might include literature that an author 
is familiar with. Typically, two coders separately review each study and extract 
data (e.g., sample size, methodology, results) and a measure of inter-rater reliabil- 
ity is computed. A meta-analysis is then typically conducted with supplemental 
analyses (e.g., forest and funnel plots) to assist in examining heterogeneity of 
results and publication bias. In some cases, a meta-analysis is not appropri- 
ate due to differing methodologies or outcome variables used in the studies 
(Cochrane Library, n.d.). Many medical systematic reviews are conducted as part 
of the Cochrane Collaboration (which includes a peer review and publication 
of the research protocol) and often included detailed information on the meth- 
odology including separate 1-2 page tables describing characteristics of each 
study (e.g., methods, participants, setting, potential sources of bias, outcomes; 
see, e.g., Merry et al., 2012). Thus, these reviews not only provide the overall 
meta-analytic results but also compile summaries of individual articles in one 
place that could be useful to researchers and practitioners. This is one practice 
that I-O psychology could consider following. 

Systematic reviews also consider the methodologies and limitations of the stud- 
ies being reviewed, which could address some limitations of meta-analysis. For 
instance, if the database of studies for a meta-analysis omitted certain potential 
moderators, populations, or other key variables, then the authors of a system- 
atic review could highlight those limitations for further research and to cau- 
tion researchers and practitioners about potential boundary conditions for the 
findings. Additionally, systematic reviews are regularly updated. The Cochrane 
Collaboration recommends that its medical reviews be examined periodically to 
determine if an update is warranted and provides guidance for doing so (Cumpston 
& Chandler, 2020). A survival analysis of 100 completed meta-analytic system- 
atic reviews determined that median survival time of a review De, the time 
at which the results had a meaningful change since the previous review was 
conducted) was 5.5 years (Shojania et al., 2007). This speaks to the impor- 
tance of (a) authors of meta-analyses examining whether their meta-analytic 
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findings have changed since publication and (b) journals being open to publish- 
ing updated meta-analyses. 


Other Configurations between Data, Methods, and Theory 


The scientific method described earlier in this chapter is the predominant 
approach for empirical research used in nearly all fields of science (Cucina et al., 
2014). I-O psychology has shifted in its approach in recent years, placing a much 
greater emphasis on theory (Cucina & Moriarty, 2015). However, there has also 
been discussion of how induction and deduction can inform scientific research. 
In our opinion, both approaches (i.e., focus on making a theoretical contribution 
and distinguishing between induction and deduction) have been misguided. In 
the next two sections, we discuss these two approaches in detail, pointing out 
how they relate to and conflict with the scientific method and how the scientific 
method should be the preferred approach in I-O psychology. 


The Myth of the “Theoretical Contribution” 


Earlier we mentioned the rise of “theory” in I-O psychology and management, 
which has been well documented (e.g., Aguinis et al., 2014; Cortina, 2014; 
Nicklin & Spector, 2016). In this section, we provide a clear definition of what 
theory should and should not be in science and explain how it is nearly impos- 
sible for authors to make a true theoretical contribution in their manuscripts. 
We make the case that by conducting research with the primary intent to make 
a theoretical contribution in our manuscripts; we are actually making our work 
less scientific. Our motivation is not to cast blame, but instead to steer I-O and 
management research onto the road of the scientific method and off of the road 
of pseudotheory. 

How Do We Define Theory? The first author began with an infor- 
mal review of the I-O psychology and management textbooks, editorial 
statements, and chapters to find a definition of “theory.” It proved difficult 
to find a clear, concise, and agreed-upon definition of what a theory is in 
the I-O psychology and management literatures. Similar observations have 
been made by Corley and Gioia (2011) and Sutton & Staw (1995). Some of 
the common characteristics of what a “theory” is include long passages of 
text containing what Cortina (2016 p. 1143) termed “revelatory original- 
ity” rather than summaries of well-replicated empirical findings. Cortina, 
Aguinis, and DeShon (2017) traced the history of publications in the Journal 
of Applied Psychology and noted that in the early 1960s, introduction sections 
were often a couple of paragraphs. By the 1980s, a couple of pages were 
devoted to theory, and since the 1990s introduction sections have continued 
to grow in order to more fully develop it. 


332 J. M. Cucina and M. A. Nester 


New “theories” are often based on case studies, common-sense, or tangen- 
tially related research. Incorporating a previously stated hypothesis or well- 
established theory is frowned upon and criticized as being “nothing new” and 
“argumentation by citation” (Ketchen, 2002, p. 586). Thus, new constructs 
are proposed and “surprising” statements are awarded the distinction of good 
theory, whereas previously supported hypotheses are not regarded as making 
a theoretical contribution (Corley & Gioia, 2011; Mintzberg, 2005). Figures 
showing complex causal models with independent, dependent, mediating, and 
moderating variables are common in theoretical work. This is despite the fact 
that the less parsimonious a causal model is the less likely it is to be true as 
pointed out recently by Saylors and Trafimow (2021) in the context of organiza- 
tional research and by Aristotle, St. Thomas Aquinas, and William of Ockham 
hundreds of years ago in the context of science in general (Kaye, n.d.). 

How Do Other Sciences Define Theory? Other scientific fields have 
much more clear and concise definitions of theory. A survey of college textbooks 
and interdisciplinary literature from other fields of sciences (e.g., astronomy, 
physics, biology) by Cucina et al. (2014) revealed a clear consensus: a theory is 
a well-replicated and strongly supported hypothesis. A similar survey was con- 
ducted for this paper using updated sources, and the definitions of hypothesis 
and theory quoted in Table 13.1 largely confirm Cucina et al.’s (2014) findings. 

In I-O psychology and management, some leading journals (e.g., Academy of 
Management Review) publish only articles which essentially propose new hypoth- 
eses and models (rather than establishing that a hypothesis is well-replicated 
and strongly supported). Leading journals in other fields of science rarely pub- 
lish articles that contain only hypotheses. For example, Nature (2015) publishes 
articles that are hypotheses “rarely, only about once a year.” Moreover, it is rare 
for a journal to publish only speculative work. We found few analogues to the 
theoretical outlets of the organizational sciences in other areas of science. One 
exception is the journal Medical Hypotheses (n.d.), which publishes “interesting 
theoretical papers.” The criteria for publication are whether or not a manu- 
script presents ideas that “are radical, interesting, and well-argued” (Corbyn, 
2010). However, Medical Hypotheses has a controversial track record and is largely 
looked down upon by the scientific community (Corbyn, 2010). 

How We Are Misdefining Theory. The theories we develop often fail to 
meet the definition of theory used in other sciences and many theories are only 
tested once, if at all (Edwards et al., 2014; Kacmar & Whitfield, 2000). Consider 
that theories in other fields of science are most often based on numerous empiri- 
cal studies that have provided considerable support for a hypothesis. Yet much of 
the pseudotheory produced in the I-O psychology and management literatures 
lacks empirical support (Edwards et al., 2014). Modern theories in I-O psychol- 
ogy and management literatures often reflect the personal viewpoints of the 
authors rather than the very empirical evidence that should be reported. This is 
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problematic as human beings have an amazing ability to create hypotheses that 
are flat out wrong. Examples exist in psychology (Lilienfeld et al., 2009), statis- 
tics (Lance & Vandenberg, 2015), logic (illogical fallacies such as assuming that if 
all S are P, then all P must be S), physics (even Albert Einstein was not infallible 
as he incorrectly hypothesized the existence of a cosmological constant; Harvey 
& Schucking, 2000; Siegel, 2013), and prescience (e.g., noting that birds did not 
succumb to the plague, medieval doctors assumed that wearing beak-shaped 
masks would make them immune to disease). 

Additionally, elegance and parsimony are considered hallmarks of true sci- 
entific theories, yet much of our pseudotheories contain multiple hypotheses, 
paths, mediators, moderators, moderated mediators, and so forth. In some cases, 
prior works (e.g., previously published hypotheses or true scientific theories) are 
cited (and miscited) with only cursory attention to how these relate to, or can 
be tested in, the current study. The typical theory produced today is unclear, 
unconcise, untested, and often untestable. Contrast this with the true scientific 
theories that exist in our field such as goal-setting theory (Locke & Latham, 
2002), validity generalization of general mental ability (Schmidt & Hunter, 
2004), classical test theory (Nunnally & Bernstein, 1994), and item-response 
theory (Hambleton, Swaminathan & Rogers, 1991). 

Sometimes researchers define “theory” as the “how” behind a particular pro- 
cess or phenomenon. In other fields of science, this is known as the mechanism 
of a process. Although the mechanism of how a particular medication works 
or selection procedure works may not be critical for practical purposes, from a 
basic science perspective these are questions that are important. However, sim- 
ply proposing a mechanism is not an adequate approach to understanding the 
“how” behind a process. Empirical work would need to be conducted to test 
the hypothesized mechanism. Thus, the “how” behind a process is a hypothesis 
in and of itself. 

How Do Many Management/I-O Psychology Researchers Define 
Theoretical Contributions? Authors today strive to make a theoretical con- 
tribution in their manuscripts. But what is a theoretical contribution? In I-O 
psychology and management journals, a theoretical contribution is viewed as 
“new and innovative ideas and insights” and “meaningfully exten[sion of] exist- 
ing theory” (Journal of Applied Psychology, 89(1), 178). It involves hypotheses, 
conjecture, stories, and imagination. Making a theoretical contribution is not 
viewed as an empirical process (in contrast to the scientific method). For exam- 
ple, consider the recommendations and inspirational messages for creating the- 
ory quoted in Table 13.2. These statements show that creating theory is much 
more art than science. 

Unfortunately, the theoretical statements that most often appear in the recent 
I-O psychology and management literature are not really theories from a scientific 
standpoint. These statements are often poorly constructed hypotheses involving 
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TABLE 13.2 Quotations showing an unscientific definition of theory by OB and 
management theorists 


Author (Year) Quote Page 


Mintzberg (2005) “We don't discover theory; we create it” 4 
“We get interesting theory when we let go of all 10 


this scientific correctness ... and allow our minds 
to roam freely and creatively — to muse like mad 
» 


“Stories and anecdotes are better than measures on 10 


seven point scales and the like” 
Davis (1971) Good theorists are “imaginative” (italics appear in 344 
original) 
“Qualitative correlations” form better theories and 323 
are more “interesting than quantitative 
correlations” 
Too much focus on the scientific method results in 328 
“the Mediocre” 
“the creative spark”” 328 
Sutton & Staw (1995) Those who are “good at theoretical” work are 380 
“dreamy” 
Klein & Zedeck (2004) “Assertion and even evidence are no substitute for 932 
explanation and interpretation” 
Shepherd & Suddaby (2017) “compelling theories are at their core compelling 60 
stories.” “theory building [can be centered] 60 


around the five key elements that inform every 
great story: conflict, character, setting, sequence, 
and plot and arc.” 


models that are too complex? to test in a single study, regardless of how well 
designed the study may be. Future researchers are discouraged from testing these 
hypotheses further because in doing so they fail to make their own new “theoreti- 
cal contribution” by proposing new hypotheses. Thus, hypotheses are often only 
tested partially in the paper that proposes them (and some are not tested at all). Asa 
result, we are creating new hypotheses and not allowing them to become falsifiable 
(one of the hallmarks of well-constructed hypotheses in science). These hypoth- 
eses often take up considerable journal page space, which is a limited resource. 
Highhouse (2014) has noted a substantial increase in the length of articles in LO 
journals. This raises a question about the value of journal pages that present ideas 
lacking evidence of support or disconfirmation. If the goal of I-O psychology and 
management research is to improve the functioning of the workplace, shouldn’t 
our journals strive to focus on publishing text that has an empirical basis? A better 
approach, as will be described below, would be to remove much of the hypoth- 
esizing and theoretical conjecture in favor of more concise and factual articles. 
The remaining pages could then be replaced with additional articles, especially 
conceptual replications of earlier works (Kepes & McDaniel, 2013). 
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What Is a True “Theoretical Contribution”? I-O psychologists’ and 
management researchers’ definition of a theoretical contribution differs greatly 
from that used in other sciences. In other fields, a theoretical contribution would 
include the results of a multi-study empirical research program that follows the 
scientific method. Locke & Latham’s (2002) goal-setting theory is an excellent 
example of a theoretical contribution, yet making this contribution involved 
a “35-year odyssey” (p. 705) with over 400 studies by many scientists (Locke, 
2007). Other examples of a theoretical contribution would include a meta- 
analysis showing a robust effect size (with small credibility intervals and no 
publication bias) or a review that convincingly shows support for a hypothesis. 
Occasionally, a crucial study that disconfirms an existing theory arises. This also 
could lead to a theoretical contribution; however, it might need to be replicated 
before rising to the level of a theoretical contribution. Thus, according to the 
scientific definition of theory and the scientific method, it would be extremely 
difficult for an author to make a theoretical contribution in a single paper. 

In fact, in other fields of science, the term “theoretical contribution” rarely 
appears in individual articles. A search of two premiere multidisciplinary scien- 
tific journals De, Nature and Science) was conducted for the phrase “theoreti- 
cal contribution” through 2020. In Nature, this phrase appeared in only 101 of 
422,374 articles. The term was almost always used to commend the work of 
someone else. Most of the hits, De, 83) bestowed the term “theoretical contri- 
bution” on someone else, often in obituaries and stories of individuals receiving 
awards. Nine articles stated that based on existing theory, something can make a 
“theoretical” contribution to a process (e.g., there might be a “theoretical con- 
tribution of the three amino-acids” [Burgus et al., 1970] in a particular reaction). 
Seven articles mentioned the phrase in announcements for other (often new) 
journals. One article complained about untested theoretical contributions and 
the last mentioned it in terms of determining authorship credit. 

The term was less commonly used in Science. Out of 297,556 articles, only 13 
included the term. Again, the phrase “theoretical contribution” was most often 
used (in 11 of the 13 hits) to commend work of someone else, especially in obitu- 
aries and announcements of awards that have been given to certain individuals. 
Only two articles actually stated that the authors made a theoretical contribu- 
tion. In an article about the planet Jupiter, Trafton and Wildey (1970) wrote that 
“The primary theoretical contribution of the study reported here has been the 
incorporation of the ammonia bands at 10 and 16 A into the model...” More 
recently, Lacour and Green (2014) authored an article on transmission of gay 
equality and stated “Our theoretical contribution is to introduce the distinction 
between active and passive contact, which are posited to produce different...” As 
an aside, this article was later retracted from Science. 

Why Making a True Theoretical Contribution in One Paper Is 
Almost Mythical Authors are encouraged to write papers that make a 
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theoretical contribution. As described above, this involves proposing a theory 
that is new and surprising, one that has not been presented in the literature pre- 
viously. However, from a scientific standpoint, a paper that truly makes a theo- 
retical contribution is almost mythical. Science is an incremental process, and 
few papers really can establish a brand new scientific theory that other scientists 
did not anticipate. Scientific theories evolve over many years and involve mul- 
tiple published and presented studies, followed by a meta-analysis or review. In 
contrast with other fields of science, our field does not view this type of evidence 
as theory (Klein & Zedeck, 2004). 

Clearly, we need to (re)embrace meta-analytic methods and other forms of 
credible evidence (e.g., systematic reviews) as a path toward establishing a theory. 
However, we also need to reconsider the notion that a theoretical contribution 
must be something completely new. This expectation has led to a proliferation 
of untested theories in I-O psychology and management (Edwards et al., 2004). 
Given the multiple steps involved in the scientific method for replication and 
designing new studies to test a hypothesis, it is often not logistically feasible 
for a researcher to create a novel hypothesis and establish it as a theory in a 
single paper, especially a primary study. Indeed, it is quite rare for a researcher 
to create a hypothesis and then find multiple existing studies showing enough 
support to establish it as a theory. Some possible exceptions exist. For exam- 
ple, Schmidt & Hunter's (1977) validity generalization of general mental abil- 
ity tests and Einstein's theory of special relativity are possible exceptions. Yet, 
these theoretical contributions, although later shown to be strongly supported, 
underwent extensive empirical testing after being introduced. Schmidt and 
Hunter encountered substantial resistance to validity generalization, but after 
many follow-up articles (e.g., Schmidt et al., 1985) it was eventually accepted by 
most of the scientific community (Schmidt, 2015). Einstein's work also received 
initial skepticism and follow-up testing (Brush, 1999; Goldberg, 1987), but now 
is widely accepted. Both theories were also based on existing experimental evi- 
dence. Validity generalization was initially based on observations that small sam- 
ple industrial validity coefficients varied widely, whereas large sample validity 
coefficients from military studies were more consistent (Schmidt, 2015). 

It also might be possible for a researcher to spend years iterating through the 
steps of the scientific method to turn his or her new hypothesis into a theory and 
then publish all of the results at once. However, this is not a wise career move; 
the researcher’s CV would have a huge hole, funding institutions and employers 
would wonder what progress is being made, and so forth. Additionally, other 
researchers would not be able to conduct independent tests and provide evidence 
that the results replicate in other contexts. Thus, this is not a viable approach. 

The Best That Can Be Done. So what is the best that can be done in a sin- 
gle paper with respect to theory? A researcher can create a hypothesis that is new 
to the literature and either test it or propose that others test it (e.g., in an outlet 
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such as AMR). After extensive testing this hypothesis could become recognized 
as a theory and the researcher could be noted for making a theoretical contribu- 
tion. Additionally, a researcher could revise an existing hypothesis (and optionally 
test it). Researchers can also continue testing previously published hypotheses, 
which may one day become theory. Hypotheses can become theories when there 
is a large body of empirical evidence providing their support. Meta-analyses and 
literature reviews summarize the results of multiple studies testing a hypothesis. 
These types of reviews (quantitative and qualitative) provide the strongest basis 
for establishing a new theory, although many in our field would incorrectly view 
them as not making a theoretical contribution. Essentially, making a theoretical 
contribution, as defined in the sciences, is most often an effort of the scientific 
community rather than an effort of one person or one paper. 

How to Use Theory Correctly. Currently, many authors develop a new 
theory to generate hypotheses for I-O psychology and management empirical 
research. However, there is another way to incorporate theory when designing 
research studies. There are many well-supported truly scientific theories in the 
basic research literature. Our field is sitting next to a scientific theory goldmine 
in other psychology journals, yet we neglect it in favor of creating our own 
untested theories. Originally, the goal of applied psychology was to apply theo- 
ries and findings from basic psychology to the real world. However, we have 
drifted away from this goal. 

Some examples of how to incorporate theory from other disciplines exist in 
the I-O related literature. For example, there is strong evidence for the theory 
of the self-serving bias in the social psychology and personality literatures (e.g., 
Dunning, Perie & Story, 1991). The essence of this theory is that when asked 
which traits are important for success, individuals have a tendency to report 
that the traits they perceive themselves as having are more important than 
other traits. Three studies have tested the theory of the self-serving bias in a 
new context, job analysis ratings, and have found that it exists in job analysis 
data (Aguinis, Mazurkiewicz & Heggestad, 2009; Cucina et al., 2012; Cucina, 
Vasilopoulos & Sehgal, 2005). In another example, based on empirical work 
encompassing over 460 datasets, Carroll (1993) developed the Three-Stratum 
Theory of mental abilities. One of these abilities, Meaningful Memory, bares 
a strong resemblance to the process of learning material in a training environ- 
ment. It involves the ability to recall, after a study period, material that involves 
meaningful interrelations (e.g., a story, a concept, a biography of an individual). 
Recent work applying this ability from Carroll’s theory shows that it is one of 
the few specific abilities that uniquely predicts training performance beyond 
general mental ability (Cucina et al., 2014). A third example resides in work by 
Carter et al. (2014) who applied the item response theory generalized graded 
unfolding model (GGUM; Roberts, Donoghue & Laughlin, 2000) from the 
psychometrics literature to scoring personality tests. 
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The Scientific Method as an Alternative to 
“Inductive” and “Deductive” Research 


In recent years, there have been calls for renewed interest in inductive research 
within I/O psychology and OB (Locke, 2007; Locke, Williams & Masuda, 2015; 
Ones et al. 2017; Spector et al, 2014; Woo, O’Boyle & Spector, 2017). This can 
be contrasted with the heavy focus on deduction and theory building in many 
academic journals (Aguinis et al., 2014; Campbell & Wilmot, 2018; Cortina, 
2014; Dilchert, 2017; Hambrick, 2007; Highhouse, 2014; Kepes & McDaniel, 
2013; Olenick et al., 2018; Ones et al., 2017; Schneider, 2018). In this context, 
induction is often viewed as making an observation that is based on empirical 
data and then developing a general conclusion or a theory. Deduction is often 
viewed as developing general conclusions or theory and testing those empiri- 
cally. In brief, induction is viewed as going from the particular to the general 
and deduction is viewed as going from the general to the particular. 

In this section, we briefly review existing conceptualizations of inductive 
and deductive research often used in I/O psychology and OB. We then pre- 
sent alternative conceptualizations of induction and deduction that are more 
closely aligned with those appearing in the philosophical and logical literatures. 
We then describe how the different conceptualizations fit into the scientific 
method. We go further by suggesting that researchers can avoid having to make 
distinctions between induction and deduction by simply following the scien- 
tific method. Aspects of both conceptualizations of induction and deduction are 
folded into and encompassed in the scientific method. Later, we explain how 
our work relates to the recent debate concerning the role of and emphasis on 
theory in I/O psychology and OB research. (See the four articles in the point/ 
counterpoint section of the Journal of Organizational Behavior edited by Nicklin 
and Spector, 2016, for an example of the debate). We conclude with recommen- 
dations for researchers. 


Deductive and Inductive Reasoning 


The roots of inductive and deductive reasoning lie in formal logic and the phi- 
losophy of science. Both induction and deduction involve using premises (e.g., 
statements, principles, evidence, observations) to reach a conclusion. In deduc- 
tive reasoning, provided that the premises are true, a valid conclusion is neces- 
sarily true with 100% certainty. For example, suppose that we are given two 
premises: “All nonprofit organizations are groups organized for a purpose other 
than generating a profit” and “G Group is a nonprofit organization.” Assuming 
the premises are true, we can validly conclude (with complete certainty) that 
G Group was organized for a purpose other than generating a profit. Logicians 
define induction as using evidence to reach a conclusion that may be true but 


346 J. M. Cucina and M. A. Nester 


that is not guaranteed to be true (Hawthorne, 2017; Skyrms, 2000). The conclu- 
sions in inductive reasoning are valid but not completely certain; instead there 
is a level of probability or confirmation associated with each conclusion. For 
example, suppose that we are given two premises: “67% of nonprofit organiza- 
tions are public charities” and “G Group is a nonprofit organization.” Based on 
the premises, we can validly conclude that there is a .67 probability that G Group 
is a public charity; thus, although it is likely that G Group is a public charity, we 
are left uncertain. 

In their comprehensive discussion of the relationship between deduction and 
induction, Colberg, Nester and Trattner (1985) showed that these two types 
of reasoning converge in terms of their forms but differ with respect to the 
certainty of their conclusions. The examples from the preceding paragraph are 
presented below to show both the convergence of the reasoning forms and the 
difference in the certainty of the conclusion. (They are presented in conditional 
form rather than in set form.) 

If an entity is a nonprofit organization, then it was organized for purposes 
other than generating a profit. 

G Group is a nonprofit organization. 

Therefore, G Group was (necessarily) organized for purposes other than gen- 
erating a profit. (deductive) 

If an entity is a nonprofit organization, then there is a .67 probability that it 
is a public charity. 

G Group is a nonprofit organization. 

Therefore (with a probability of .67), G Group is a public charity. (inductive) 

In the first example above, G Group must have been organized for purposes 
other than generating a profit because of its membership in the set of nonprofit 
organizations, all of which were organized for purposes other than generating 
a profit. This example represents a basic form in deductive logic, called “modus 
ponens,” in which a necessary conclusion is drawn about an individual based on 
information in a universal premise. It begins with the premise “if p then q.” If it 
is affirmed that p is true, then it follows that q is true. In this example, a neces- 
sary conclusion is drawn about an individual based on information in a universal 
premise. In the second example, there is no universal premise. Instead, the first 
premise says that if an entity is a nonprofit organization, then there is a .67 prob- 
ability that it is a public charity. 

The valid conclusion is stated with a probability of .67, which means that it is pos- 
sible but not certain that G Group is a public charity. As Colberg et al. (1985) explain: 


When logicians state that an inductive conclusion is never certain, they 
mean that there is no identity between the premises and the conclusion. 
In a deductive conclusion there is such an identity. This is the same as say- 
ing that in a deductive conclusion, if the premises are true and the schema 
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is correctly constructed, then the conclusion cannot be false, whereas in 
an inductive conclusion, even if the premises are true and the schema is 
correctly constructed, the conclusion can still be false. The falsity of the 
conclusion in an inductive schema is compatible with the premises and with 
the schema. Thus, a deductive conclusion is always necessary, never proba- 
bilistic, and an inductive conclusion is always probabilistic, never necessary. 


(p. 682) 


As a consequence of the identity between premises and conclusion in deductive 


‘ 


arguments, they “... are usually limited to inferences that follow from defini- 
tions, mathematics, and rules of formal logic” (Internet Encyclopedia of Philosophy, 
2016). By contrast, scientific research tries to go beyond deductive argument by 
observing natural phenomena. When these observations are about sets that can- 
not be known in their entirety, observations are usually made about a sample 
from the phenomenon being investigated. Any general conclusion drawn from 
such samples is an inductive, probabilistic conclusion. The statistical tools that 
enable scientists to estimate, for example, a population value from sample statis- 
tics have been developed mathematically, that is to say deductively. The use of 
these estimates to describe the population in question, however, represents an 
inductive exercise because the estimates are based on incomplete information 
about the population. 

In the organizational science and I/O psychology literatures, deduction is 
often defined as applying a general principle or rule to a specific case, and induc- 
tion is often described as observing a series of specific cases and inferring and 
formulating a general principle from the specific cases. Colberg et al. (1985) 
traced this misconception of inductive reasoning to early psychometric research 
by Thurstone (1938), who defined the term as “find[ing] a rule or principle for 
each item in the test.’ It has also made its way into the research methods lit- 
erature, whereby the process of first observing data and then making a general 
statement (e.g., a hypothesis or theory) is defined as inductive research and the 
process of first making a general statement (especially a hypothesis) and then 
observing data is defined as deductive research. This way of defining inductive 
and deductive research is incompatible with definitions used in logic and the 
philosophy of science, and it obscures the fact that all scientific research involves 
both induction and deduction. 


The Scientific Method Incorporates the Process of Going 
From Particular to General and Vice Versa 


The previously described distinction between inductive and deductive research 
in terms of general to particular and particular to general research can be rendered 
moot by adoption of the scientific method, which incorporates both processes. 


348 J. M. Cucina and M. A. Nester 


The process of going from particular to general appears in several steps in the 
scientific method. A researcher could observe a particular phenomenon in step 1 
De, make an observation) and then construct a research question and hypothesis 
in steps 2 and 3 which later lead to a general conclusion in step 8 De, form a 
theory). For instance, a researcher might begin with the observation that scores 
on certain measures in a battery she administered are related to leadership suc- 
cess. The finding could be used to form more general research questions and 
hypotheses concerning the constructs measured in the battery. The working 
hypothesis could then be tested in a new study. After repeated replication and 
meta-analysis, it might be possible to establish a general conclusion and theory 
about the constructs in question and leadership success. Unless the conclusion is 
true with deductive certainty, it is considered an inductive conclusion. 

The process of going from general to particular is also incorporated in the 
scientific method, especially in steps 3 to 6 De, hypothesizing, predicting, and 
testing). A researcher with a conjectural statement could begin at step 3 of the 
scientific method and design empirical studies to test that conjecture and make 
predictions about the findings for particular datasets and studies. 

It is also possible to start with a general corroborated scientific theory and then 
ask research questions and create hypotheses extending the theory to particular 
new settings. For example, a researcher might start with a theory that is established 
in the basic psychological research literature (e.g., social psychology) and see if it 
applies to a particular organizational issue. Alternatively, a researcher might begin 
with a theory that applies to a certain population and then test its validity in another 
population (e.g., examining whether the five-factor model of personality seen in 
Western samples applies to a newly studied applicant pool in a different country). 

Overall, we believe that organizational researchers do not necessarily need 
to make the distinction between going to and from particular to general when 
conducting research. Instead, they can simply adopt the scientific method as it 
is used in other fields. Full adoption of the definition of the scientific method 
used in other fields would allow organizational researchers to make and publish 
research going from the particular to the general, without special calls for induc- 
tive research. It would also simplify the instruction of students. Learning the 
distinction between general and particular or inductive and deductive would no 
longer be needed. In addition, there would be greater consistency in how the 
scientific methods are taught in organizational sciences courses and courses in 
other areas of science. 


The Scientific Method Incorporates the Philosophical Definitions 
of Induction and Deduction 


In this section, we describe how the philosophical definitions of induction 
and deduction are incorporated in the scientific method. Induction deals 
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with probabilities and it appears clearly between steps 5 and 6, during which 
a researcher makes a probabilistic determination (based on statistical signifi- 
cance and effect size) as to whether the hypothesis is supported. It also appears 
clearly in step 7, when a field of scientific researchers makes a determination as 
to whether there is a high enough probability that a hypothesis is true for it to be 
considered a theory. Bayesian statistics, whose creator was one of the principal 
contributors to the field of inductive logic (Fitelson, 2006), can play a key role 
in these determinations. Indeed, validity generalization (a scientific theory) uses 
Bayesian statistics (Schmidt & Hunter, 1977; Schmidt & Raju, 2007; Schmidt 
et al., 1979), and some authors have noted the ability of Bayesian statistics to 
synthesize prior knowledge, accept null hypotheses, and state theories (Jebb & 
Woo, 2015; Kruschke, Aguinis & Joo, 2012). 

There are also some striking similarities between the outcomes of Bayesian 
statistical analyses (e.g., a probability) and the results of expectancy table analyses 
used in the personnel selection literature. For instance, given the meta-analytic 
operational validity of .66 for general mental ability tests in medium-complexity 
jobs, an individual in the top 25% of scores on this test has a 57.6% probability of 
being in the top 25% on the criterion (compared to 3.1% for individuals with a 
test score in the bottom 25%).° These types of probabilities are commonly used 
to predict risk in actuarial and medical settings, to create weather forecasts, and 
even predict earthquakes. However, in all instances, the conclusions involve a 
probability; even if the probability can be stated with extreme precision (e.g., 
the 95% confidence interval for the probability of 57.6% in the above example is 
55.7% to 59.6%)°, the conclusions about an individual case are still uncertain and 
involve a probability. Essentially, at a global level, all empirical research is induc- 
tive and none is exclusively deductive because the researcher is always making 
an inference based on incomplete information and the conclusion can only be 
probabilistic. 

Although all empirical research in psychology is inductive, there are exam- 
ples of truly deductive research in these fields. Mathematics is a field that makes 
extensive use of deduction, especially in mathematical proofs and theorems 
(which are essentially theories based on deduction). Consider the mathemati- 
cal proof for the correction of a correlation coefficient between a predictor (p) 
and a criterion (c) for criterion unreliability. Using a deductive process, it is 
possible to go from the equations for a partial correlation coefficient and a vari- 
ance decomposition to the equation for the correction for criterion unreliability. 
Assuming that the premises are true (e.g., the formulas for partial correlation 
and variance decomposition are correct, the algebraic manipulations used in 
the derivation are correct, measurement error is random and uncorrelated with 
the predictor and criterion), the conclusion (i.e., the formula for correction for 
unreliability is toc /afrelo ) must be true. This type of reasoning is prominent in 
the psychometric literature. In many ways, psychometricians could be called 
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theoretical psychologists and the derivations and proofs for classical test theory, 
item-response theory, and generalizability theory are examples of deductive 
reasoning. 

There are also instances in which either of the philosophical definitions of 
induction and deduction can appear in the scientific method. Consider step 1 of 
the scientific method (making an observation). Whether induction or deduction 
is involved depends on the nature of the observation itself. An observation that 
entails complete information on the population of interest could be the basis for a 
deductive conclusion. For instance, an organizational scientist might observe that 
all of the employees in an organization who were rated unsuccessful were trained 
at training center X. If the entire population was studied here, a valid deductive 
conclusion would be that if employee Y was rated unsuccessful, then employee 
Y was trained at training center X. An observation that would be the basis for an 
inductive conclusion would be one in which the organizational scientist observed 
that 80% of the employees rated unsuccessful were trained at training center X. 
A valid inductive inference would be that if employee Y was rated unsuccessful, 
then there is a .8 probability that employee Y was trained at training center X. 


Conclusion 


The scientific methodology of I-O psychology and management research is cur- 
rently drifting away from the scientific method and toward an embrace of pseu- 
dotheory. Our research culture is impeding adoption of the scientific method. 
Researchers are motivated to make theoretical contributions in their papers, yet by 
doing so they have adopted an incorrect viewpoint of what theory is. According 
to the scientific method, making a theoretical contribution in a single primary 
study is impractical if not impossible. If the research literatures of I-O psychology 
and management are to truly make a contribution to scientific knowledge, major 
changes are needed. (Re)Adopting the scientific method provides a sound and 
established basis for further advancing I-O and management research and making 
better contributions to organizations. It also provides a framework for how data, 
methods, and theory are necessarily related in scientific research. 
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The low level of statistical power in many psychological studies is also unfortunate. 
Thirty years ago, Cohen (1992) lamented that there was little increase in statistical power 
of psychology studies since the first edition of his power handbook (Cohen, 1969) was 
published. Low statistical power continues to be an issue in the literature as the Open 
Science Collaboration (2015) reported a median sample size of 54 cases for the 97 origi- 
nal psychological studies it replicated. The mean replication effect size was a correlation 
of .197. Thus, on average, the authors of the original studies were attempting to detect 
a correlation of .197 with only 54 cases, which equates to a power of .30 according to 
G*POWER (Gaul et al., 2007; Faul et al., 2009). 

Traditionally, a laboratory notebook consists of a bound book with blank pages in which 
a researcher documents their empirical research and analyses. Writing is often done by 
hand in ink and the pages cannot be removed inconspicuously. This allows for a diary or 
journal of the researcher activities and can be made available to other researchers for 
inspection. Computer-based versions of laboratory notebooks are also used. 

This might be due to the perceived complexity of human behavior. Oftentimes 
we hear that organizational behavior is the study of people (in organizations) and 
that an individual human’s behavior is complex, thus requiring complicated psycho- 
logical models. However, psychology is not all that different from the other sciences 
in regard to the magnitude of empirical findings, the precision of measurements, 
the consistency of results, and the accuracy in predicting individual-level outcomes 
(Hedges, 1987; Meyer et al., 2001). Indeed, very few sciences can predict individual- 
level outcomes with complete accuracy. Consider the difficulty a meteorologist faces 
predicting the weather on any given day with complete accuracy. Actuaries predict- 
ing whether or not an individual will file an insurance claim and credit bureaus pre- 
dicting individuals’ creditworthiness also have difficulty predicting future events with 
complete certainty. Even in the laboratory sciences, difficulties exist. Most chemical 
reactions do not result in all molecules/atoms reacting (i.e., the “actual yield” of a 
most chemical reactions is less than the “theoretical yield”). Although there are many 
causes of an individual’s behavior, there are also many causes of the behavior of an 
individual molecule, the weather, the economy, and so forth. In other fields, it is 
often helpful to study the effects of an independent variable on a dependent variable 
using parsimonious models and theories. We suggest that the same process be used 
by organizational researchers. Attempting to create a model that explains everything 
using every possible independent variable leads to a model that is difficult to test and 
to validate. It also introduces the possibility of redundancy and overlap in the inde- 
pendent variables (e.g., does the second independent variable relate to the dependent 
variable because it is a proxy for the first independent variable or are both measures 
of the same construct?). 

This incorrect definition persists in the psychometric literature and appears in Carroll’s 
(1993) treatise and McGrew’s (2009; McGrew & Evans, 2004) reviews. Although using 
the incorrect definition of induction, Carroll (p. 238) noted the work of Colberg and 
her colleagues on general-to-particular inductive tasks. 

These values were obtained using Hunter et al’s (2006) reanalysis of Hunter’s (1980) 
meta-analysis analysis and syntax for computing expectancies from Cucina, Berger & 
Busciglio, (2017). 

This is based on the expectancies (obtained using Cucina et al’s, 2017, R. syntax) for the 
95% confidence interval about the point estimate of .66 (.63 to .69 using the sample 
size of 12,933 reported by Hunter, 1986, and applying the corrections for unreliability 
and range restriction to the uncorrected upper and lower bounds of the confidence 
interval). 
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